docs(submitqueue): combine predictor into Scorer - #682
Open
behinddwalls wants to merge 5 commits into
Open
Conversation
## Summary ### Why? Speculation ranks paths on the scorer's price, which only reflects the change (lines, files) and ignores what the run has since observed — a passed build, a failed build, batch state. The predictor design needs a written contract before the implementation lands. ### What? Adds doc/rfc/submitqueue/outcome-predictor.md: scorer vs predictor, the four YAML factors and what their values mean, which path results count as evidence, and rejected alternatives. Links it from doc/rfc/index.md. ## Test Plan Docs-only change.
This was referenced Sep 7, 2026
## Summary ### Why? The RFC's rejected-options list used implementation shorthand and included alternatives already explained elsewhere, making the design tradeoffs difficult to evaluate without branch context. ### What? Keep the four alternatives reviewers are likely to raise and state each rejected option, why it fails, and the selected design. Remove redundant and scorer-specific entries.
behinddwalls
force-pushed
the
preetam/outcome-predictor-rfc
branch
from
September 7, 2026 23:54
451805e to
3227132
Compare
behinddwalls
marked this pull request as ready for review
September 8, 2026 00:08
## Summary ### Why? The outcome predictor RFC described failed-path evidence and ranking differently from the path-set identity and best-first behavior, while older speculation RFCs still assigned dynamic pricing to the scorer. ### What? Document that prediction applies to unresolved dependencies, failed all-succeeds evidence applies once, neutral factors preserve exact scorer prices, and factors must be finite. Update the existing speculation RFCs to consistently describe predictor-based ranking and state-aware pricing.
Document per-queue factor inheritance and the shared batch/path-set snapshot consumed by best-first ranking.
behinddwalls
force-pushed
the
preetam/outcome-predictor-rfc
branch
from
September 8, 2026 00:57
01a9de2 to
9696e21
Compare
sbalabanov
requested changes
Sep 10, 2026
sbalabanov
left a comment
sbalabanov
left a comment
Contributor
There was a problem hiding this comment.
- Is predictor a first class abstraction (i.e. an interface) or implementation detail of the generator?
- Let's discuss as an alternative to have all the decisions inside scorer. Scorer is called for each batch anyways? Why can't be made evidence-aware?
- The proposed implementation of the predictor is somewhat hard to read. Can we frame it as a linear regression model with defined features and configured weights? What in the proposal does not fit into a definition of LGM?
Fold ranking into one Score(ctx, batch, paths). Evidence is the scorer; heuristic/composite are the nested base. Frame the revision as a configured logit-linear model.
behinddwalls
changed the title
Collaborator
Author
|
Agreed on all three — the stack now does (1)+(2) as one Scorer, and (3) is documented on that impl.
The rewrite is in this PR; impl / wiring / demo are #683, #684, #686. |
sbalabanov
approved these changes
Sep 12, 2026
| @@ -0,0 +1,126 @@ | |||
| # Outcome Scorer | |||
Contributor
There was a problem hiding this comment.
Predictor is gone, file name still specifies it
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Why?
Ranking only needs one number per unresolved batch. A sibling Predictor factory duplicated the per-queue seam and made evidence look like a second public extension. The review asked for one ranking interface, framed as a logit-linear model.
What?
Scorer is the ranking extension:
Score(ctx, batch, paths). Evidence is the default impl wrapping a nested base (heuristic or composite). YAML istype: evidenceplusfactorsplusbase. The revision islogit(p') = logit(p_base) + Σ w_i x_iwith configured weights (w_i = log(factor_i)), not a fitted GLM.Test Plan
Docs-only change.
Issues